Papers with DeepSpeed solution
Full Parameter Fine-tuning for Large Language Models with Limited Resources (2024.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) require massive GPU resources for training. |
| Approach: | They propose a parameter-efficient optimization that fuses the gradient computation and parameter update in one step to reduce memory usage. |
| Outcome: | The proposed method reduces memory usage to 10.8% compared to the standard approach. |